On-line learning of lexical items and grammatical constructions via speech, gaze and action-based human-robot interaction

In order to be able to understand a conversation in interaction, a robot, has to first understand the language used by his interlocutor. A central aspect of language learning is adaptability. Individuals can learn new words and new grammatical structures. We have developed learning methods that allow the humanoid robot iCub to robot can learn new lexical items by interaction with the human and consolidation of its autobiographical memory. Then, based on these open class words, the robot can bootstrap the acquisition of novel grammatical structures in real-time. Finally, we demonstrate how human gaze can be monitored, and could be used in order to reduce referential ambiguity inherent in such learning conditions. These learning capabilities are demonstrated in a collection of videos. 1. Learning of ”open-class” word through Human Robot interaction, using an autobiographical-like memory In robotic systems based on human-inspired robot task learning [1], significant attention has been allocated to the mechanisms that underlie the ability to acquire knowledge from and encode the individual’s accumulated experience [2], and to use this accumulated experience to adapt to novel situations [3]. In this context we consider research on autobiographical memory (ABM) and mechanisms by which ABM can be used to generate new knowledge [4]. Figure 1: Physical interaction between the iCub and an human. Both are interacting with the ReacTable (the blue table). Behind the robot, its internal representation is displayed on a screen. The objective of the current research is to demonstrate how an autobiographical memory system, coupled with mechanisms for detecting and extracting regularities can be used to construct a progressive hierarchy of spatial, and temporal relations that provide the basis for learning and executing shared plans. The mixed-initiative dialogue between the Robot and the Agent is done as described in the Figure 1 and 2, by a simple oral interaction. RTable iCub Motor Cmd